DOCOHENT BESDME 

EA 005 352 

I 

Pearson^ Robert H. 

An Instructional Management Guide to Cost Effective 
Decision Making • ^ 
Instructional Science Associates ^ Madison^ Wis, 
TR-73-11-1 
73 

69p, 

MF-$0,65 HC-$3-29 

*C6st Effectiveness; ^S'Decision Making; Educational 
Objectives ; ^Educational Planning; Management 
Systems; Models ; Resource AllocationvS; ^Systems 
Analysis 

^Instructional Management 



This study attempted to identify the differences in 
cost-effectiveness analysis methods in order to select the most 
appropriate one as a decision-assisting aid for instructional 
management. The study is organized to explore and identify what 
cost-effectiveness methods are being used, identify and examine the 
differences among these methods in an analytical framework, and 
determine the criteria to be used in selecting the most appropriate 
cost-efiectiveness method for instructional decisionmaking. The study 
results suggest that cost-effectiveness analysis provides answers by 
a systematic process that is reproducible as well as aacessible to 
critical examination. It is believed that the use of instructional 
cost-effectiveness analysis will enhance the possibility that a 
decision will result in the selection of the best alternative. 
(Author/DN) 



ED 083 711 

AUTHOR 
TITLE 

INSTITUTION 
REPORT NO 
PUB DATE 
NOTE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



FILMED FROM BEST AVAILABLE COPY' 



.:.C'Pv 



rO iniL i-;:, IK,.;....-;.; of'f .... 

UfvDI'M A(,Mf F^n fJ:S .--ilr- f:.'. ti; if, 

QUI»F:S PE:nMI?,;^tCif. ')f 'Ml. COPYMIOHT 
OWNLf 



©Copyright 1973' by Instructional Sciences Associates. No part of this 
publication may be reproduced or u^ed in any form without penrdssion of 
the publisher. Printed in the United States of America and published by 
Instruct! :inal Science Associates ^ 5 South Yellowstone Drive, Madison 
Wisconsin 53705, 



CO 

o 



U.S. DEPARTMENT OF HEALTH. 
EOUCATtON&WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 
THIS DOCUMENT HAS BEEN RE PRO 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORtGiN 
ATING IT. POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE 
SENT OFFICIAL NATIONAL INSTITUTE. OF 
EDUCATION POSITION OR POLICY 



TR73-11-1 



CD 



\ 



AN INSTRUCTIONAL MANAGEMENT GUIDE 
TO COST EFFECTIVE 
DECISION MAKING 

by 

Robert H. Pearson, Ph.D. 
University of Wisconsin-Madison 




'•?*en MISSION TO reproduce this COPY- 
RIGHTED MATERIAL HAS BEEN GRANTED BY 



INSTRUCTIONAL SCIENCE ASSOCIATES j^^^f^i^^^^^ 

TO ERIC AND ORGANIZATIONS OPERATING 
UNDER AGREEMENTS WITH THE NATIONAL IN- 
STITUTE OF EDUCATION. FURTHER REPRO- 
\ DUCTION OUTSIDE THE ERIC SYSTHM RE- 

. V . j * , QUIRES PERMISSION OF THE " COPYRIGHT 

\ \ \ i \ OWNER." 



FOREWORD 



. Education currently consumes a vast amount o£ 
resources, and it appears almost certain that it will 
consume much more in the future. Since the systems an- 
alysis process makes possible the evaluation o£ alter- 
native courses o£ action for the achievement of instruc- 
tional goals and objectives within resource constraints, 
those in instructional management are enthusiastic over 
its potential as a viable base for decision-making. 
Cost-effectiveness analysis, one aspect of systems an- 
alysis, has been shown to aid in the better use of re- 
sources. There are, however, a variety of cost-effec- 
tiveness analysis methods and confusion has existed as 
to the respective advantages and disadvantages of each 
method, this guide, therefore, is directed towards the 
analysis of cost-effectiveness analysis itself from the 
point of view of instructional management. This approach 
is so that the most appropriate instructional cost-ef- 
fectiveness analysis method may be selected as an aid to 
decision-making. 



Robert H . Pearson 

University of Wisconsin-Madison 
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PREFACE 



There are a variety o£ cost-effectiveness analy- 
sis methods ■ and these methods differ in approach and 
precision." This guide will attempt to present the most 
applicable cost-effectiveness analysis method as a deci- 
sion-assisting aid for instructional management. Thus, 
the approach taken . is to: (1) explore what cost-ef- 
fectiveness methods are being used; (2) examine the 
differences of these methods in an analytical frame- 
work; and, (3) suggest the most . appropriate cost-ef- 
fectiveness analysis method for instructional manage- 
ment decision-making. 

Due to the complexity of the area of systems/ 
cost analysis, the study, will, be limited to the subject 
of cost-effectiveness analysis itself. In addition, the 
emphasis, will be on the performance of the cost-effec- 
tiveness analysis by someone other than the instruction- 
al management decision-maker , as a service in support of 
the decision-making process. The decision-maker, in this 
case, .might be: (1) a dean or department head of an 
academic institution; . (2) a training manager or super- 
visor of industry or government; or, (3) a superinten- 
dent, principal, curriculum director, or program direc- 
tor of a school system. 

Although the guide includes considerable detail- 
ed information involving the documentation of procedural 
steps in various cost-effectiveness analytical techniques, 
it is still less than would be required for an analyst to 
use all aspects of the procedure. An attempt was made, 
therefore, to limit the amount of detailed description 
to the minimum information requirements for the perfor- 
mance of a cost-effectiveness analysis. 



R.H.P. 
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ABSTRACT 



The purposa of the study is to identify the differ- 
ences in cost-effectiveness analysis methods so that the most 
appropriate one could be selected as a decision-assisting aid 
for instructional management. 

The principal phases of the study are organized to ex- 
plore and identify what cost-effectiveness methods are being 
used; to identify and examine the differences among these 
methods in an analytical framework; and, to determine the 
criteria to be used in selecting the most appropriate cost- 
effectiveness method for instructional decision-making. 

The emphasis of the study is on -the performance of cost 
effectiveness analysis as a service in support of the decision 
making process by someone other than the instructional manage- 
ment. The descriptive detail provided in the study is such 
that an analyst with moderate experience could perform the 
indicated activities . 

The results o£ study suggest that cost-effectiveness 
analysis provides answers by a systematic process which is re- 
producable and accessible to critical examination. On the 
other hand, cost -effectiveness analysis cannot be a substitute 
for judgment in decision-making by inductive reasoning which 
is fully documented. It is believed that the use of instruc- . 
tional cost-effectiveness analysis will enhance the possibil- 
ity that a decision will result in the selection of the better 
(or best) alternative.* 
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INTRODUCTION 



Education the world over is a vast enterprise and one 
that continues to grow. Estimates vary concerning the pre- . 
cise cost of education and training in the United States; 
,"It's like asking how many trees there are in Russia," com- 
mented a training program authority who has attempted to deter- 
mine exact accounting several times (Kl.einschrod , 1967 , p. 18). 
It is suggested that approximately 64-million persons in the 
United States are involved in some aspect o£ education and 
training and the annual cash outlay is estimated to be an ex- 
penditure o£ $90 billionl (Goodman, 1973, overleaf). 

With 1:he expenditures of resources of such magnitude 
and with the competition for th?6se resources by other needed 
public enterprise, educators must search for ways to optimize 
utilization of the ins tructional 2 dollar. One possible way to 
atta.in maximum output from our instructional resources is to 
increase the effectiveness of our materials, methods, facili- 
ties , etc . 

This purpose has little relation to that once vaulted 
"cult of efficiency" that sought to bring business methods to 
bear on inefficient schools. The purpose of sensible economic 
practices in education has less to do with efficiency than with 
effectiveness . It is a question. . .of turning out the highest- 
quality product . possible for the funds, talent, and time expended. 
[U.S. Congress, 1970, p. 24]. 



Another way to get more from instructional resources is 
to consider alternative methods of reaching a goal through sys- 
tems analysis . . ^ 

The systems approach is an "economic" approach to educa- 
tion in the best sense of that word. It provides a rational 
method of using a giver set of resources to provide a system cap- 
able of achieving a given set of objectives [Springer, 1967, p. 58] . 



/ •''About 50 percent of most municipal budgets are allocated to 
public education (Bumes, 1967, p. 39). 

^For the purpose of this guide, the term /'instruction" will be 
used to refer to both education and training. 
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To some, systems analysis is characterized solely by auto- * . . 
matic data processing, techniques and is dismissed as being 
modern gadgetry. This may not be true, for most systems an- 
alysis formulations are indistinguishable from descriptions 
of the scientific method. Consequently, systems analysis 
■roots reach back past Roger Bacon to Aristotle, and not, as 
some believe, only to the RAND Corporation and the Department 
of Defense of the 1960^s (Oettinger , 1968 , p. 77). The newer 
approaches to systems analysis, however, attach more impor- 
tance to the: (1) establishment of quantifiable objectives 
and alternatives; (2) identification of the alternatives 
cost and benefits; and, (3) provision for analysis time 
(McGivney, 1969, p. 32). It must be emphasized that systems 
analysis is most valuable as an analytic tool to aid in in- 
structional management decision-making, rather than as a deci- 
sion-maker itself (Oettinger, 1968 , p. 77), 

Cost-benefit analysis and cost-effectiveness analysis 
make up that aspect of systems analysis which is concerned 
with the selection of alternatives based on consideration of 
the best use of the available resources to reach required ob- 
jectives. This analysis of resource usage requires synthesiz- 
ing knowledge about the student, the instructional setting, ob- 
jectives , facilities , instructional method, staff, standards 
of achievement and other related factors. This systems/cost- 
effectiveness analysis approach has the goal of establishing 
improved relationships between these interacting elements so 
that the total instructional enterprise is performing more 
effeciently and at a higher level of excellence (Freedman, 
1969, p. 32) . . 

The need for and value of cost-benefit analysis and 
cos t-effectiveriess analysis has been recognized in public and 
private education. It was reported by the Commission on Educa- 
ticmal Technology that in improving learning, cost and. cost- 
benefit relationships were the areas of greatest uncertainty 
(Carpenter, 1970, p. 5). A. prominent private committee re- 
commended that: 

School systems must employ continuously the result of 
cost-benefit and cost-effectiveness analysis in order to allocate 
effectively the resources available to education and to distin- 
guish among programs of high and low priority [Committee for 
Economic Development, 1968, p. 13]. 

In addition. President Nixon \ms quoted in the Washington Post 
as saying that for education, the 1970 's will become the "Age 
of Accountability," and that one aspect will be the judgment of • 
school programs relative to cost-effectiveness (Wentworth, 1970) . 



3/4 



It would seem that the current position on system/cost- 
effectiveness analysis in support of instructional management 
decision-making is that many are enthused over the potentional 
o£ it (McGivney, 1969, p. 31) --but who will perform it and how 
itwiirbe done are more difficult considerations. Systems/ 
cost analysis personnel might come from the following areas: 

• Government. Those individuals as- 
sociated with the Department of 
Defense and their prime contractors 
who hope to transfer their profes- 
sional and scientific capabilities 
from defense to education-oriented 
activities (Lieberman, 1968, p. 24) . 

. • Business . Independent consultants 
with no vested interest in hardware 
or a product (Herbert, 1967, p. 45). 

• Education. Educators and admini- 
strators responsible for planning 
(Heinich, 1969, p. 330). 

Attaining a process by v/hich cost-effectiveness analy- 
sis may be applied to instructional decision-making, however, 
is a more formidable task. One difficulty is that • although 
much has been reported on the advantages of instructional 
cost-effectiveness, only a limi^ted number of '^how to do cost- 
effectiveness*^ studies have been published (Fisher, 1967, p. 69) 
Another problem is that the usefulness of xost-ef f ectiveness 
analysis, is not fully understood by many individuals. This 
lack of understanding is due to the apparent coldness of the 
tool as a measure of decision-making (Banghart, 1969, p. 206) . 

There are a variety of cost-effectiveness analysis me- 
thods in use today which differ in their respective advantages 
and limitations (Seller, 1969, pp. 1-2) . As new approaches 
develop, these methods constantly undergo subtle, but signifi- 
cant, changes (Kazanowski, 1968, p. 163) . Although most of 
these new approaches are primarily mathematical in nature^ a 
few are directed toward a better treatment of problems that 
cannot be handled by purelv quantitative methods (Quade, 1968, 

p. 242) . : ' 

With the apparent lack of understanding, limitations, 
and wide differences of approaches in cost- effectiveness analy- 
sis, the following question seems appropriate. How can this 
analytical tool, which was developed as a decision-assisting 
aid in the selection of technical systems, be used by instruc- 
tional management increasing student achievement (Freedman, 19 69 
p. 31)? The attempt to provide an answer to this question is 
the purpose of this guide. , 
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BACKGROUND. OF COST-EFFECTIVENESS ANALYSIS 



Cost-eff ec tivdness analysis is not completely " new • To 
some extent, it has always been part of industrial and govern- 
ment planning and procurement • As recently as a few years ago, 
however, the use and understanding of government cost-effec- 
tiveness analysis was inadequate for two major reasons: (1] 
all elements of cost were not included in the analysis; and 
(2) the probabilistic nature of system effectiveness was not 
recognized (Herd, 1965 , ppT 79-80) , By the late 1950''s it 
became obvious that technical improvements in governmental 
programs were not always accompanied by improvements in total 
program performance and effectiveness. As a consequence, a 
foundation for more systematic and objective methods of mea- 
suring the effectiveness of alternative courses of action for 
military systems was developed and implemented in 1961 by the 
Department of Defense (Jakobsberg> 1966, p. 38). The next 
major step occurred when the President directed the introduc- 
tion of an integrated planning-prbgramming-budgeting system 
into the Executive Branch of the ^Federal Government; this 
system included cost-effectiveness analysis as a tool for re- 
source allocation decisions . 

Today, the cost-effectiveness approach has been adopted 
by many other activities of governm.ent as well as organizations 
of private industry (Seller, 1969, pp..vii-l). According to 
Jarrett: 

The significance of cost-effectiveness analysis today 
stems from the increasing sophistication of our methods in relat- 
ing cost to effectiveness as compared with methods available only 
a few years ago. It has been referred to as '^qualified common 
.sense", which actually is close to what cost-effectiveness is all 
about [Jarrett, 1967, p. 8] . 

PURPOSE OF COST-EFFECTIVENESS ANALYSIS . 

The purpose of cost-effectiveness analysis is to provide 
the decision -maker with data on the cost (resource consumption) 
and the probably effectiveness (quality of results to be a- 
chieved) ^for each of several alternatives, among which Jie must 
choose one course of action, .Cost-effectiveness analysis in- 
volves clarifying .the relationships between these two factors 



(cost and effectiveness) so that the decision-maker can strike 
the best balance between them . Cost-effectiveness analysis 
is not a substitute for judgment, experience, and common 
sense. It is a method for the systematic examination of all 
the subjective and objective data available (lleymont, 1965 , 
p. 1). 



PERFORMANCE OF COST-EFFECTIVENESS ANALYSIS 

The current trend in the performance of cost-effective- 
ness is the application of the method by all levels of deci- 
sion-making, in a variety of functions, and for both short- 
range and long-range planning (Heuston ^ Ogawa, 1966, p. 243). 
The analysis itself may be performed by the decision-maker, 
by a specialist within the decision-maker's organization, or 
the responsibility for cost-effectiveness analysis may be del- 
egated to a specialized service organization (McGivney, 1969, 
p. 31). The latter approaches require close cooperation and 
coordination between the cost-effectiveness analyst primarily 
responsible for the study, the decision-maker requesting the 
study, and each of the other individuals providing information- 
al inputs (Fields, 1966, p. 517) . At best, cost-effectiveness 
analysis functions as a tool used in providing the decision- 
maker with an analytical foundation for making sound objective 
choices among the various ways a problem might be solved or 
an objective met (J^rrett, 1967 , pp. 7 fi 9). 



DEFINITION OF COST-EFFECTIVENESS ANALYSIS 

The definition of cost-effectiveness analysis varies, 
for the methodology is not precise enough to mean the same 
thing to all practitioners in the field (Seiler, 1969, p. 1). 
Broadly defined, it is an analytic study designed to assist 
a decision-maker in identifying a preferred choice among pos- 
sible alternates (Quade, 1969, p. 1). The choice of alternates 
involves two approaches: (1) fixed-cost and flexible effec- 
tiveness approach ; and , (2) fixed -effectiveness and flexi- 
ble cost approach (Kazanowski , 1968, pp. 126-127). Several 
varieties of a more precise definition are graphically present- 
ed in Figure 1 to illustrate the emphasis different authori- 
ties have placed on specific nomenclature. 

Attention should also be given as to what is not cost- 
effectiveness analysis. Kazanowski calls this the "maximum- 
effectiveness -at -minimum-cost fallacy" and defines this mis- 
conception as follows: 
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THE BASIC OBJECT OF ^OST-EFFECTIVENESS ANALYSIS IS 
TO ISOLATE THE ALTERNATIVE OR COMBINATIONS OF ALTER- 
NATIVES THAT GrVE(S) : 



1 . The mi nimi zi ng of 
- costs subject to 
some objective 
constraint - 


or 


a maximizing of seme 
physical measure of 
the objective out- 
put subject to cost 
constraint (Ouade, 
1969, p. 2) . 


2. The greatest 

objective effect- 
iveness for a given 
expected cost 


or 


a given expected object- 
i veness for the 
least expe.cted cost 
(Seiler, 1969, p. 1 ). 


3. The greatest 

effectiveness for 
any given cost 


or 


a required or chosen 
degree of effective- 
ness for the 1 east ~~ 
cost (McGi vney and 
Nelson, 1969, p. 105). 


4. The mos t 'producti ve 
alternative if the 
costs are equal 


or 


the least costly 
. al ternati ve that 
produces equal 
benef i ts . 


If the alternatives are of unequal costs and produce 
unequal benef i ts , cost-effectiveness analysi s wi 1 1 
provi de ■ i nf ormati on for comparing- costs and benefits 
(Office of the Assistant Secretary of Defense, 1971 
p. 5). 



Figure 1. Variation in Cost-Effectiveness 
Analysis Definition 

ERIC 
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Many references can be found in which the statement is 
made directly or indirectly that cost-effectiveness is a tech- 
nique by which one determines that alternative which yields the 
maximum effectiveness as a minimum cost. Or, in other worcs, 
that alternative is sjught by which the effectiveness is max- 
imized, while the cost is minimized. In reality the attempt to 
find such an alternative is doomed to failure at the onset be- 
cause such an alternative does not exist. Hitcli and McKean state 
''Actually, of course, it is Impossible to choose that policy whic 
simultaneously maximizes gain and minimizes cost, because ^.here i 
no such policy." [Kazanowski, 1968, p. 160]. 

DIFFERENCES BETWEEN COST-EFFECTIVENESS 
AND COST-BENEFIT ANALYSIS 

Confusion also exists concerning the difference bet- 
ween cost-effectiveness analysis and cost-benefit analysis 
(Fisher, 1967, p. 66). The principal differences in these 
two methods are the elements of output and of time-phasing . 



Output 

The term ''cost-effectiveness" usually applies to situa 
tions in which it is difficult to quantify the alternate out- 
puts in terms of dollar values. Cost-benefit usually consi- 
ders only the monetary worth of outcomes. 



Time Phasing 

The term "cost-effectiveness" is usually used to selec 
the most appropriate choice from the predicted alternative 
outcomes; tne term "cost-benefit^' is most often applied in 
selecting the most appropriate choice from the calculated pre 
sent (McGivney and Nelson, 1969, p. 105). 

Cost-effectiveness analysis is specifically directed to 
problems in which the output cannot be evaluated in market prices 
but where the inputs are substitutable at exchange relationships 
developed in the market [Niskanen, 1969, p. 18]. 

SUBJECT APPROPRIATENESS FOR COST-EFFECTIVENESS ANALYSIS 

Certain prerequisites should be met before a subject 
is appropriatjs for cost-effectiveness analysis (Kazanowski, 
1968, p. 114). The primary appropriate prerequisite is that 
the subject should have two characteristics: first, and most 
important, the subject must be relevant; second, but less 
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important, it should be capable of being measured both in re- 
quired resources and outcome. These two characteristics often 
appear conflicting. The most relevant phenomena or events are 
often very difficult to measure, the least relevant are usually 
easy to measure (Niskanen, 1969, p. 20) . Cost-effectiveness 
analysis should be performed only when the following appro- 
priate prereq^iisi -^s are met: 

• The correct problem must be recog- 
nized • 

• Realistic objectives that bear a 
functional relationship to the prob- 
lem must be determined (Jakobsberg, 
1966, p, 38) . 

• Common goals, objectives, or mis- 
sions of the alternative must be 
identifiable and at least theoret- 
ically attainable. 

• Constraints for bounding the problem 
must be discernible. 

• Alternative means of meeting the 
goals must exist (Kazanowski, 1968, 
p, 114) . 

The meeting of these requirements is not always simple; 
consequently, if ther^"^ is no freedom of choice in problem solv- 
ing, cost-effectiveness analysis is not appropriate (Jarrett, 
1967, p. 8) . 



RELATIONSHIP OF COST-EFFECTIVENESS TO SYSTEMS ANALYSIS 

In order to place cost-effe&tiveness in proper perspec- 
tive, Lt is first necessary to place it in relation to the sys- 
tems a lalysis process. Cost-effectiveness analysis is not a 
methoc for measuring either cost or effectiveness after the 
fact, for its modern applications are concerned with the pre- 
dicti n and evaluation of an alternative's worth (English, 1968, 
p. 7). The term ^'systems analysis'^ is used to distinguish this 
broad r analysis from a narrow cost-effectiveness aspect of 
analysis . Cos t- effectiveness analysis is that aspect of sys- 
tems analysis which emphasizes the systematic investigation o£ 
the decision-maker's alternatives relative to objective criter- 
ia, costs, and effectiveness comparison, in addition to the 
other trade-off factors associated with alternative selection 
(Quade, 1969, p. 3) . Figure 2 graphically illustrates this 
emphasis . 
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BROADLY DEFINED 

PHASES OF 
SYSTEMS ANALYSIS 



GOAL/OBJECTIVES/ 

REQUIREMENTS 

DEFINITION 



I 



ALTERNATIVE 
DEVELOPMENT 



I 



ALTERNATIVE 

TRADE-CrF 

STUDIES 



I 



ALTERNATIVE 
SELECTION 



4- 
4- 



VARIOUS AIDS IN THE 
DECISION-MAKING 
PHASE 



RELIABILITY 
ANALYSIS AND 
OTHER TECHNIQUES 



COST-EFFECTIVENESS 
ANALYSIS 



CONSTRAINT 
IDENTIFICATION AND 
OTHER TECHNIQUES 



Note: After Heaton, 1969, p. 35 



Figure 2. Relationship o£ Cost Effectiveness 
to Systems Analysis 

ERIC 
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■1 . ■ ■ . ■ 

It can be seen, then, that the difference between cost- 
effectiveness analysis and systems analysis is a matter of emphasis. 
If the emphasis is on finding significant differences in the costs 
or resource requirements among the available alternatives for 
carrying out some specified task, the analysis is generally re- 
ferred to as a cost-effectiveness analysis. The systems analyst, 
on the ether hand, is likely to be forced to deal with problems 
in which the difficulty is deciding what ought to be done, not 
simply how to do it. System analysis thus puts greater emphasis 
on the suitability of the task and the augmentation of alternatives. 
■ In both system-analysis and cost-effectiveness work it is impor- 
tant to note, however, that after an appropriate evaluation one 
will be in a much better p'osition to make a decision, even though 
he may not find the decision easier to make [Bell, 1964, pp". 1-2], 

CATEGORIES OF COST-EFFECTIVENESS ANALYSIS 

The confusion over the role cost-effectiveness analysis 
plays in the systems analysis process seems to arise from the 
fact that some practitioners perceive cost-effectiveness analy- 
sis to be the systems analysis process itself ; whereas, others 
see the process o£ cost-effectiveness as being related, to. and 
a sub-set of the total systems design process • As rioted above, 
both interpretations of the role of cost-effectiveness analy- 
sis could be appropriate, depending upon the time and condi- 
tions that the analysis requires. There are, however, three 
categories of cost-effectiveness analysis which, in practice, 
are not clearly distinct. These categories are: (1) system/ 
component configuration study; (2) system comparison study; 
and, (3) suprasystem Comparison study (Heuston ^ Ogawa, 1966, 
p. 245). The relationship of these categories is exhibited in 
Figure 3 and a detailed description follows. 

System Configuration/Component Study 

In this category the emphasis is on the selection of the 
particular configuration, or characteristics, of a single >sys- 
tem with different components. For example, a closed-circuit 
instructional television system ^versus an open-circuit instruc- 
tional television system. This 'category of study is probably 
the one most often performed. 

Cost Treatment. Typically, system/component configura- 
tion studies have the following characteristics relating to the 
treatment of costs: 

' « Detailed costs are obtained on cill 
components . • 
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VALUE 
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CATEGORIES OF ANALYSIS 



VOCATIONAL 
RETRAINING 



INSTITU- 




ON-THE-JOB 


TIONAL 




TRAINING 


STUDY 







SUPRA- 
SYSTEM 
COMPARISON 
STUDY 



SELF- 




AUTO- 


CONTAINED 




TUTORIAL 


CLASSROOM 




STRATEGY 


STRATEGY 







SYSTEM 
COMPARISON 
STUDY 



CLOSED- 
CIRCUIT 
TELEVISION 
RECEPTION 



OPEN- 
CIRCUIT . 
TELEVISION 
RECEPTION 



VIDEOTAPE 
RECORDER 
TELEVISION 
RECEPTION 



SYSTEM 
CONFIGUR- 
ATION/ 
COMPONENT 
STUDY 



± 



iTELEVISION 
RECEIVER A 



[ 



TELEVISION 
RECEIVER B 
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TELEVISION 
RECEIVER C 



Figure 3 . Cost-Effectiveness Categories of Analysis 



13 



• Various mathcmat.i c:]l models arc 
used to simulate :ind optimize dif- 
ferent possible configurations of 
the components . The results yield 
the predicted overall costs of the 
various alternatives under study. 

• Costs are compiled by summing the 
yearly costs over the life cycle 
o£ the systems rather than by com- 
piling time-phased costs. 

Effectiveness Treatment. As in the treatment "of cost, 
the effectiveness measure used in the system configuration/com- 
ponent study will be detailed in terms of capabilities and per- 
formance. In most cases, effectiveness will not be a simple 
overall measure which ultimately would determine alternative 
selection (Hatry, 1969, pp. 45, 47-48). 

It should |be noted that trade-off analysis at the detail 
level of selecting subcomponents is classified as value analy- 
sis rather than cost-effectiveness analysis (Kazanowski, 1968, 
p. 153). An example of this type of analysis would be the se- 
lection of the least expensive instructional television receiv- 
er that complies with given specifications. 



System Comparison Study 

Here, the emphasis is on comparing two or more systems 
for the same objective. The emphasis is on intersystem, not 
intrasystem, analysis. It is presumed that each competing sys- 
tem has already been sub-optimized as to its configuration, 
through, the system/configuration component study discussed pre- 
viously. An example of this type of study would be to compare 
the merits of a programmed-ins tructional system versus an auto- 
tutorial system (vAere each has been instructionally validated) , 

Cost Treatment. Typically, the system comparison study 
has the following characteristics relating to system, costs: 

• Costs are generally required in 

. less detail than in system/compo- 
. ment configuration studies where 

the emphasis was on components of 

competing similar characteristics, 

• The specific spread of costs over 
the life cycle of the system is 
usually ignored, or is treated as 
a secondary problem. 
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Effectiveness Treatment. The effectiveness measure is 
basically quite similar to that used in system/component con- 
figuration studies. 

Suprasystem Comparison Study 

This third cost-effectiveness study category is less 
frequently performed as a strictly cost-effectiveness activity 
Activities at this level of analysis are usually described as 
cost-benefit. The problem here is to assess the merits of al- 
ternatives of widely differing capabilities and cost elements . 
An example would be to analyze the cost-effectiveness of com- 
peting instructional systems such as institutional study as 
opposed to on-the-job training. 

Cost Treatment. In suprasystem comparison studies, 
costs are each time-phased to the extent possible, to provide 
•estimates of annual resource requirements (Hatry, 1969, pp. 48 
§ 50) . 

Effectiveness Treatment. The determination of a mea- 
surement of effectiveness .is especially difficult for those 
cases where the systems are experimental, the objectives are. 
of unequal priority, and the output varies in type and relia- 
bility (Heuston § Ogawa, 1966, p. 245). 

The problem of the analysis of differing costs and capa 
bilities of alternatives requires the development of some mea- 
sure of structure for comparing these alternatives on a some- 
what equal basis (Department of Defense, 1971, p. 5). 

PROCESSES OF COST-EFFECTIVENESS ANALYSIS 

In addition to the confusion from the differences of 
cost- effectiveness analysis categories, differences in the pro 
cess of the analysis itself has compounded the lack of under- 
standing. Yet, in all the variations, two processes are per- 
formed; (1) the procedural process; and, (2) the applica- 
. tion process . 



Procedural Process 

This cost-effectiveness analysis process can be defined 
as the documentation or writing of instructions on how to per- 
form the analysis in a step procedural sense. This type of 
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process is also known as model building; i.e., a conceptual 
.mock-up or detailed paper-pencil description of the actual 
analysis to be performed at some later time. 



Application Process 

The second cost-effectiveness analysis process refers 
to the application of the procedural process by applying a 
particular model to a real -world practical situation (Speagle, 
1969, p. 22) . 
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STANDARD PROCEDURE FOR COST-EFFECTIVENESS ANALYSIS 



The chief value o£ cost-effectiveness analysis is its 
broad ability to provide a general fraiaeworlc to assist in tha 
solution of problems in a wide range of topical areas. Cost- 
effectiveness analysis differs from many other decision-assist- 
ing tools which are limited to accepting only certain classes 
of problems. 

The very fact that cost-effectiveness analysis can as- 
sume characteristics dictated by a unique objective and res- 
pond to the category, level, and process of -^.vialysis as re- 
quired provides a lack of uniformity in the analysis itself 
(Jakobsberg, 1966, p. 37). In addition, the apparent lack of 
uniformity in most examples of cost-effectiveness analysis, 
and the lack of consistent documentation, has led some persons 
to question the merits of this type of analysis as an aid to 
decision-making (Kazanowski, 1968, p. 113). 

On the other hand, certain authors and authorities in 
the field indicate that a complete "how- to -do -it'^ document, or 
a "cookbook,'' would be impossible to prepare (Jakobsberg, 1966, 
p. 37). This may be because, at the current stage o£ develop- 
ment, cost-effectiveness analysis is more an art than a science 
(Fisher, 1967, p. 70). Nevertheless, it is suggested that the 
majority of approaches to cost-effectiveness analysis have cer- 
tain common functional elements which may be synthesized into 
a standard procedural approach. These steps are illustrated 
in Figure 4 and include the following inputs from others, anal- 
ysis, and outputs: 

• STEP 1: Review definition of the desired gen- 
eral goals and specific objectives that the de- 
sign is to meet or fulfill. 

• STEP 2; Review identification of the parameters 
or requirements essential for the attainment of 
the desired goals/objectives. 

• STEP 3: Review development of the alternatives 
for accomplishing the goals/objectives/require- 
ments . 

• STEP 4: Establish an evaluation criteria (mea- 
sures), for cost and effectiveness, that relate 
alternative capabilities to the goals/objectives/ 
requirements . 
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Figure 4. Cost-Effectiveness Analysis Standard Procedure 
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• STEP 5: Determine the cost o£ alternatives 
in terms o£ evaluation criteria. 

• STEP 6: Determine the capabilities or effec- 
tiveness of the alternatives in terms of eval - 
uation .criteria. 

• STEP 7: Select fixed-cost or fixed -effective- 
ness approach. 

• STEP 8: Create an alternatives versus criter- 
ian array. 

f STEP 9: Analyze the merits of al ternatives , 
© STEP 10: Analyze the uncertainty factors* 

• STEPll: Document the rationale, assumptions, 
and analyses underlying the previous steps and 
submit findings (Pearson, 1973, p. 34). 

The following discussion of these procedures suggests guide- 
lines to be followed for a standardized approach to the analy- 
sis of system/cost-eff ectiveness . Although the presented 
steps are in an order in which they would usually be performed, 
changes in the sequence are acceptable and should fit the need 
of the subject under evaluation (Kazanowski, 1968, pp. 115- 
116) . 



STEP 1: REVIEW DEFINITION OF THE GOALS AND OBJECTIVES 

Before cost-effectiveness analysis can be .considered , 
the problem must be identified and the goals/objectives must 
be defined, at least in a general nature. Obvious as this 
may seem, the perception of the problem that needs solving, 
and the establishment of goals and objectives, are very often 
complex and involved tasks for which there are no techniques 
and tools besides "logical thinking" (Jakobsberg, ^ 1966, pp. 37- 
38). Without such an identification of goals/obj ectives there 
is no framework for structuring the subsequent evaluations. 
The following are several points to consider in reviewing the 
general goals/objectives definition. 

If goals/objectives specified are in too general terms, 
the constraints established for bounding the evaluation are 
often only the product of the analyst.' In addition, care must 
be exercised not to identify the goals/objectives in such a 
manner as to bias the evaluation by including requirements, of 
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such specific nature that they exclude potential candidate 
alternatives from consideration. 

On the other hand, care must also be taken not to make 
goals/objectives too specific or they will limit the scope of 
possible candidate alternatives by implicitly defining alter- 
native concepts rather than just the desired goals/objectives. 
A potential danger always exists in that the goals/objectives 
originator may specify a goal/objective that is unattainable 
by means of current technology. 



STEP 2: REVIEW THE IDENTIFICATION OF THE PARAMETERS" OR 
REQUIREMENTS 

The basic purpose of identifying and defining the gen- 
eral goals/objectives is to aid in the identification of re- 
quirements essential to attaining the defined goals/objectives. 
The goals/objectives and requirements should be identified and 
specified within the required parameters. This specification 
is used to further reduce the possibility of biasing, the cost- 
effectiveness analysis. The confusion brought about by close 
relationship between goals/objectives and requirements is fur- 
ther compounded by the variety of categories of cost-effective- 
ness studies (i.e., suprasystem comparison study and system 
configuration/component study) . The. need for detailed speci- 
ficity can be illustrated by the following simplified example 
of the general goal/objective of eliminating poverty of migrant 
agricultural workers by vocational skill training. One re- 
quirement would be the identification of current labor market 
vocations needed so that detail job performance training spe- 
cifications could be formulted* The resulting requirement spe- 
cification might detail tool . or machine operator competencies 
needed, as well as conditions and level of performance expect- 
ed after training completion. The major factors for consi- 
deration when reviewing the identification of goals/objectives 
and requirements, and the development of specifications are 
as follows, 

• Verify the establishment of a relationship that 
converts, goals/objectives "into some unit of mea- 
sure for both cost and effectiveness. For ex- 
ample, number completions at a given level of 
achievement. 

• Be alert for errors of commission^, which are 
just as important as errors of omission. If 
gOEls/obj ectives that are not necessarily es- 
sential are identified as requirements , they 
can strongly prejudice the subsequent cost-ef- 
fectiveness evaluation (Kazanowski, 1968 , pp. 
117-119) . 
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STEP 3: REVIEW DEVELOPMENT OF THE ALTERNATIVES 

As in defining the goals/objectives and identifying the 
requirements, developing valid alternative means or courses of 
action that can satisfy the goals/objectives is a mental exer- 
cise for which no current scientific technique exists (Jakobs- 
berg, 1966, p. 39). This activity is to create two or more 
alternative concepts of ways to achieve the goals/ob j ectivcs 
within the defined parameters or specifications. Frequently, 
considerable imagination and creativity are required to de- 
velop effective competitive candidate alternatives. The re- 
sults of cost-effectiveness analysis can be no better than the 
conception and development of attractive candidate alternatives 
(Kazanowski, 1968, p. 120). 

There are several problems associated ^;ith the review 
of the alternative development. The most serious problem or 
source of defects is attention bias. This is frequently 
caused by a cherished belief or an unconscious adherence to 
what we might call a "party line." This attention bias may 
result in the unwarranted favoring of a particular alternative 
by the analyst (Quade, 1969, p. 8) . Another problem is the 
degree of detail associated with available information on al- 
ternatives. Since the purpose of cost -ef fee tiveness analysis 
is to aid the decision-maker in deciding which alternative 
should be selected, specific details on possible alternatives 
are generally lacking. Too little detail usually results in 
a large variance in estimated alternative effectiveness and 
cost. On the other hand, to require that the candidate al- 
ternative be designed in detail, before being evaluated, would 
defeat the basic purpose and value of cost-effectiveness an- 
alysis. In addition, the development of new and novel alter- 
natives might be discouraged by detail design requirements. 
This factor would tend to favor existing or more traditional 
alternatives. The balance between inadequate detail and ex- 
cessively rigorous specificity should be maintained. 



STEP 4: ESTABLISH COST AND EFFECTIVENESS EVALUATION CRITERIA 

The criteria specifies: (1) the measure of cost; (3) 
the measure of effectiveness; and, (2) the relationship bet- 
ween them. All will be used in the selection of one alternative 
from the several alternatives. The measure of cost must be 
consistent with the overall framework of the resource alloca- 
tir»n proHlem; e.g., net future total cost o£ ownership for 
a certai^i number of years. The measure of effectiveness 
should express the extent to which the goals/objectives and 
requirements are being accomplished. The relationship between 
the cost and effectiveness must be expressed in functional 
terms; e.g., in a mathematic equation^ graphic display, or 
model (Bryk, 1965, p. 1). 

ERIC 
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The selection o£ appropriate and adequate cost and ef- 
fectiveness evaluation criteria is based on judgment and ex- 
perience. The omission of significant criteria could readily 
invalidate the results of a cost-effectiveness analysis. The 
inclusion of numerous criteria to be on the safe side, how- 
ever, can result in mental paralysis when the optimum alter- 
native selection is being made. The greater the number of 
criteria, the greater the probability that even unlikely can- 
didates will excel in some criterion, making alternative sys- 
tem selection difficult. One simple test of the adequacy or 
completeness of cost and effectiveness evaluation criteria is 
to question whether one alternative could excel in most of 
the criteria established and still not be selected as the pre- 
ferred choice. If the answer is affirmative, important cri- 
teria could be missing. Considerable insight into the sub- 
tleties of the goals/objectives and requirements is usually 
necessary for the establishment of meaningful alternative 
evaluation criteria (Kazanowski, 1068, p. 123). Certain eval- 
uation criteria of cost and effectiveness suggested by a var- 
iety of authors are shown in Figure 5. 

Develop Measurers of Cost 

The choice of a particular alternative for accomplish- 
ing the goals/objectives implies that certain specific re- 
sources will be consumed and could no longer be used for other 
purposes. These are the costs. In future time period analy- 
sis most costs could be indicated as money expenditures, but 
their true measure should be expressed in terms o£ the values 
of alternative opportunities which are precluded because of 
this expenditure (Quade, 1969, p. 4). One of the most diffi- 
cult problems in establishing cost evaluation criteria is the 
determination of appropriate cost measures. The choice of the 
proper cost measure involves both difficulties and controversy 
within the broad categories of time-phased analysis (cost dis- 
tributions over extended time periods) and non- time-phased 
analysis. 



Cost Measures for Non-time-phased Analysis, For the 
purpose of non- time-phased analysis (such as applied to the 
system/component level configuration or system level compari- 
son study) the following three cost measures are commonly sug- 
gested. 

• Initial cost (including research and develop- 
ment and ^initial investment, to achieve the as- 
sumed goals/objectives) plus X years of operat- 
ing costs. The choice of what X should be is a 
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Figure 5. Cost and Effectiveness Evaluation Criteria 
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problem but is is most commonly taken as five 
and occasionally ten years. 

• Adjusted initial cost plus X years of operat- 
ing costs. The difference here is only the 
use of adjusted initial cost in which case the 
costs are adjusted to take into consideration 
the estimated residual values for each alter- 
native. The residual-value problem occurs most 
frequently when comparing choices between al- 
ternatives which either: (1) could be used 
for additional goals/objectives; or, (2) the 
life-cycle of alternatives differs widely. 

The latter situation is the mpst often cause 
for initial cost adjustment. 

• The above first or second cost measure, but 
discounted in some manner. Discounting has 
been used in order to perform two functions: 
(1) as an adjustment for the increasing time 
period uncertainties; and, (2) to indicate 
economic impact. 



Cost Measures for Time-Phased Analysis. For the supra- 
system comparison studies, there are three general cost mea- 
sures suggested to be used in examining each alternative. 
These are: 

• Annual funding/budget requirements • 

• Cumulative funding/budget requirements . 

• Present expenditure worth at a selected dis- 
count rate. 

The problem in all of the above measures is to determine how 
far out the' study should be carried in. order to consider all 
time periods pertinent to the purpose of the study. At some 
point the effect of the subsequent years would have negligible 
effect upon today's decisions. 

There is probably no ideal answer to the question of 
the cost measure to be used, in either time-phased or non- 
time-phased analysis. No single cost measure incorporates 
all pertinent elements and it may well be desirable to use 
more than one measure to give the proper cost perspective 
(Hatry, 1967, pp. 64-67). 
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Develop Measurers of Effectiveness 

Of all the components or constituents of cost-effec- 
tiveness analysis the cost-measure concept has found the 
least acceptance as a valid measurement criterion. Less dif- 
ficult was the acceptance of the effectiveness-measure cri- 
terion (B^^ckner, 1967, p. 83) • Proper effectiveness criteria 
selection, however, is one of the most formidable problems of 
cost-effectiveness analysis . If the goals/objectives are not 
trivial, then real world facets are generally complex and have 
widespread implications ; consequently the effectiveness cri- 
teria may be broad and numerous . If the scope of the goals/ 
objectives is reduced, usually so is the number of significant 
Criteria. Even if the scope of the problem is significantly 
reduced, it is virtually impossible to reduce the total cost- 
effectiveness analysis to a single easily evaluated effective- 
ness criterion (Kazanowski, 1968, p, 152). On the other hand, 
the more narrow and fewer in number the measure of effective- 
ness criteria become, the more limited will be the eligible 
alternatives to be considered. It is often desirable, there- 
fore, to broaden a cos t -effectiveness analysis study to in- 
clude more effectiveness criteria in order to insure a wider 
range of alternatives (Breckner, 1967, p. 58). Well defined 
and explicit measures of effectiveness are difficult to devel- 
op and rare. Typically, these measures are replaced by two 
devices: (1) performance identifiers; and (2) standards. 



Performance Identifiers. These effectiveness measures 
relate to tlae administrative control of functions where output 
is inadequately defined. Usually, performance measures are 
specified on an activity basis to give an estimate of work 
performed. The performance identifier tends to be a device to 
measure work performance and is designed to detect the varia- 
tions in work performance levels. In fact, however, perfor- 
mances may liave little relation to the final output. 



Standards. These effectiveness measures are much broad- 
er than performance identifiers. Government procurement ac- 
tivities are characterized by the use of standards, which may 
be of two types: technical performance standards, and gener- 
alized standards. Considerable overlap exists, however, and 
many problems are common to both. 

• Technical per f oi^aance standards exist through- 
out government and industry and are needed to 
give a vital measure of uniformity and int^r- 
changeabili ty of parts , components , assemblies , 
and processes. For example, both the govern- 
ment Bureau of Standards and the industrial 
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Underwriters Laboratories devplop operational 
and performance standards for electrical items.. 
These standards are observed by those manu- 
facturers who wish to market products in the * 
United States. 

• Generalized standards are identified to dis- 
tinguish them from technical, and are used 
when specific output and effectiveness mea- 
sures are difficult to determine. For exam- 
ple, certain accrediting organizations devel- 
op guidelines and s tandards for the evalua- 
tion and certification of public schools, aca- 
demic institutions , and libraries . These 
standards are observed and maintained by 
those wishing accreditation (Teitz, 1968, p. 
309) . 

Unique to instructional design tliere are three broad 
categories of instructional effectiveness measures: (1) stu- 
dent/group-oriented, (2) school -oriented; and (3) community - 
oriented . 



Student/Group Oriented Measures. These effectiveness 
measures refer to academic achievement change (grades] result- 
ing from an improvement in the instructional program. 

School -Ori anted Measures. These effectiveness measur- 
ers are chara(j:terized by three types . 

• The change in number dropping out of school as 
a consequence of these achievement changes in 
a given school population in terms o£ changes. 

• The number selecting the various available 
courses of study . 

• The number and quality of those graduating (as 
indicated by scores on standardized tests] . 



Community-Oriented Measures. These effectiveness mea- 
sures are characterized by two types. 

• Th^ average expected lifetime earnings poten- 
tial of persons with different levels of edu- 
cation (economic factors) . 
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• The reduced relationship between parental socio- 
economic level and individual achievement indi- 
cating increas ed equality of educational oppor- 
tunity (social change) (Abt Associates, Vol. I, 
1967, p. 3) . 

Although not applicable in all circumstances, the following 
ground rules may be observed in selecting appropriate measures 
of ef f ectivenes s . 

• Do not force a quantification measure of effec- 
tiveness where no sound analytical basis for 

it exists. 

• Some effectiveness criteria will be quantifi- 
able, but many will not be. 

• Nonquantif iable effectiveness criteria pose 
no particular difficulty in analysis when 
appropriate . 

• When the analysis involves a mixture of quan- 
titative and qualitative effectiveness criter- 
ia, the cost-effectiveness analyst must exer- 
cise caution in gauging the impact of tlie qual- 
itative criteria. For this reason, quantitative 
criteria are usually preferred, and tlie result 
is interpreted in terms of the qualitative cri~ 
teria (Jakobsberg, 1966, pp. 39-40). 

A final consideration is to choose a measure of effec- 
tiveness which serves as a sufficient input to tlie next level 
of decision-making (Niskanen, 1969, p. 30). 

A methodological consideration discussed in Step No. 4 
dealing with measures of both cost and effectiveness is that 
no single criterion measure that is appropriate for all goals/ 
objectives in known. Even if a single criterion measure is 
adopted for all goals/objectives, the present inability to 
quantify relationships suggests the use of a multidimensional 
S':ale for measuring results (Packer, 1068, p. 235). 



STEP 5: DETERMINE COST OF ALTERNATIVES 

Cost and effectiveness of alternatives must be deter- 
mined in order to rationally allocate resources . Costs are 
considered easier to determine than effectiveness because 
dollars, manpower, and materials are homogeneous and measurable. 
Accurately predicting future costs is not simple, hoxs^ever , 
because consideration must be given to unce::'tainty of cost 
estimates, the unpredictability of inflation, and to the 
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probable time-value of money due to discounting. Otiier fac- 
tors that could also be included are tlie cost of opportunities 
lost and tlie effect of uncontrolled consequences of the out- 
comes (Packer, 1968, p. 247). 

The problem of dollar-cost estimating often assumes so 
much importance in any cost-effectiveness analysis that the 
usefulness of the basic concept of cost (as it relates to the 
analysis) is lost. As a result, accounting/budget costs may 
take on an importance ^vhich may not be justified. It should 
be remembered that costs as prediction of the future can only 
be estimates and we always estimate future values on the basis 
of past experience. As a consequence, it is natural that re- 
cords of past experience are reviewed in order to project fu- 
ture costs. Since the accounting/budget costs records usually 
include some aspects of actual past costs, a critical review 
is to ensure that past under or over cost expenditures will 
not prejudice the current cost estimate (English, 1968, p. 77) 

The pertormance of Step No. 5 should deal with cost 
analysis for cost-effectiveness and similar analysis. The 
term ''cost analysis," in one form or other, is also used for 
those involved in budgeting, accounting, and cost-control pur- 
poses. Although somewhat related, thv:-se methods are not con- 
sidered here (Hatry, 1069, p. 44). In cost-effectiveness 
analysis, the development of highly detailed cost breakdowns 
are not required (Novick, 1967, p. 103). It should be also 
noted that the preparation of the cost estimate is usually per 
formed by someone other than the cost-effectiveness analyst. 
It is axiomatic that the estimate cannot be better than the 
statement of the goals/objectives/requirements on which the 
cost estimate is based. These statements, however, sometimes 
reflect the cost estimator's own particular biased interests 
(Novick, 1963, p. 102). In the creation of a cost estimate 
in accordance with the goals/objectives/requirements, the 
cost-effectiveness analyst must verify that the cost estima- 
tor has provided the following cost data. 



Identify Cost Categories 

These categories cover the source of major costs (Herd, 
1965, p. 81). Measures of cost (developed in Step No. 4) 
should be indicated, as well as the following elements: 

• Matter-energy, information, and time. 

• Human effort (student time and effort, teacher 
time and effort, administrative or technician 
time and effort) . 
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• Monetary costs (short run and long run cap- 
ital and operating activities) (Miller ^ 
Rath, 1969, pp • 21-24) . 



Identify Resource Constraints. 

These constraints usually result in the indication that 
liighly aggregated costs are useless; that costs must be bro- 
ken down into relatively small units whicli will permit a vari- 
ety of cost analysis approaches. 



Calculate Costs. 

Each category identified should be considered on an 
appropriate time basis as well as in total. (Although the 
alternative total cost may be within the total resources 
available, the annual expenditures may not be compatible with 
the budget or funding pattern.) 



Identify Possible Cost Variances. 

A single cost estimate does not indicate unknowns re- 
lative to other costs provided by others. Unknown factors in 
cost data elements should be identified for subsequent uncer- 
tainty analysis . 



Verify Completeness. 

Insure compliance of cost data elements with a check- 
list (Herd, 1965 , p. 81) . 

Essential to the validity of all cos t - effectiveness an- 
alysis is the availability of appropriate, accurate, and timely 
cost estimates. It is true that a poor cost -ef f ectiveness an- 
alysis can misuse even the best cost data, but even the best 
of analysis cannot do much v;ith inadequate cost data (Hatry, 
1969, p. 55) . 



STEP 6: DETERMINE CAPABILITIES OR EFFECTIVENESS OF ALTERNATIVES 

Once the effectiveness evaluation criteria have been 
determined (Step No. 4, pp. 25-27, above) the next step is to 
expres s the capabilities of tlie alternative systems in terms 
of the evaluation criteria. To save time, this is often per- 
formed concurrently with Step No. 5. Quantitative expressions 
of capabilities are preferred when they arc available but qual- 
itative expressions are acceptable when they are not (Kazanowski, 
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1968 , p. 129) • As is cost estimation, the estimates o£ the 
capabilities o£ alternative systems is dependent upon infor- 
mation and data supplied by someone, other than the cost-effec- 
tiveness analyst. Although this is both an advantage as well 
as a disadvantage, confidence in the accuracy of alternative 
system capability may be lacking and may be subject to the 
biased interests of the estimator (Quade, 1969, p, 10). 



STEP 7: SELECT FIXED-COST OR FIXED-EFFECTIVENESS APPROACH 

In assessing alternatives, the procedure may now take 
either one of two analytical approaches: 

• A desired level of effectiveness may be speci- 
fied, and the analyst seeks the most economi- 
cal way to achieve it, or 

• The level of expenditure may be specified and , 
the analyst explores the effectiveness offered 
by the alternative capabilities (Breckner, 1967, 
p. 43) . 

The choice of specifying either the fixed-cost or the 
fixed-effectiveness analysis approach oftentimes is the option 
of the analyst. This choice is important because, in most 
cases, real-world problems cannot be adequately dealt with in 
such a simple form as optimizing either cost or effectiveness. 
This is because in most studies unique measures of cost and 
measures of effectiveness can only be obtained by directing 
the analysis on a very small aspect of the total goals/objec- 
tives and requirements. The results of such an analysis is 
rather superficial and seldom reflects the real-world goals/ 
objectives and requirements from which the decision-maker must 
select the best alternative. The selection of one approach, 
therefore, may more closely reflect the real-world options 
actually available to the decision-maker, whereas the selec- 
tion of another approach might reflect options which are not 
available. Another factor in the choice of one approach over 
the other may also yield economies of analysis, depending on 
the availability, quality, and validity of data to be utilized 
in the cost-effectiveness analysis itself . The choice of 
fixed-cost of fixed-effectiveness approach is least affected 
by the constraints imposed by the decision-maker, therefore, 
the choice of approaches is vital to the integrity of the an- 
alysis. 



Fixed-Cost Approach 

#» 

A basic fixed-cost approach is first, the development 
of alternative systems that can compete in meeting the goals/ 
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objectives/requirements (Step No. 3) in competition for the 
given resources. Second, the number of components of each 
alternative that can be developed, procured, and operation- 
ally implemented within the fixed resources, is determined. 
Finally, the alternative which satisfies the goals/objectives/ 
requirements to the greatest extent is identified as the pre- 
ferred choice. 



Fixed-Effectiveness Approach 

A basic step in the fixed-effectiveness approach is 
first, development of a measure of the effectiveness express- 
ed in terms of the requirements identified in the prior Step 
No. 2, Then, alternatives are developed to meet the goals/ 
objectives as in Step No. 3. The number of components of 
each alternative necessary to attain the goals/objectives is 
. determined and, the costs and penalties incurred by each a] - 
ternative are estimated. Finally, the alternative that ex- 
hibits the least aggregate cost penalty is identified as the 
preferred choice. 

It is suggested that if irjflexible constraints are im-- 
posed on the resources available , the use of the fixed-cost 
approach is indicated. Conversely,, if inflexible constraints 
are imposed on the effectiveness required, the use of the fixed- 
effectiveness approach is indicated. In the absence of other 
guidelines, the presence of both a large number of significant 
' measures of effectiveness criteria and a small number of mea- 
sures of cost criteria usually indicates that the fixed-effec- 
tiveness approach should be used (Kazanowski , 1968 , pp. 126- 
128) . 

Note that in using either the fixed-cost and or fixed- 
effectiveness approaches to evaluate alternatives, the mea- 
sure of criteria (either the cost criteria or the effective- 
ness criteria) must be held constant so as to have a common 
basis for comparison. Methods in which both the measures of 
cost and the measures of effectiveness arc allowed to vary 
are to be avoided because there would then be no common basis 
for comparison (Barfoot, 1963, p. 3) . 

STEP 8: CREATE ALTERNATIVE VERSUS CRITERIAN ARRAY 

Ideally, it would be desirable to examine the cost and 
effectiveness evaluation criteria of the alternatives in the 
real world rather than in the simulated world of cost-effec- 
tiveness analysis. For many obvious reasons, such as exces- 
sive expense and/ or the unavailability of the alternatives, 
O this is not possible (Blumstein, 1969, p. 33). Two different 
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analytical techniques o£ conducting cost=ef f ectiveness eval- 
uations within either the fixed-cost or fixed-effectiveness 
approach are often encountered. These are: (1) the tabular 
display technique; and, (2) the model technique. 



The Tabular Display Technique 

The tabular display technique is used when the alter- 
natives are being evaluated by measures of cost, or measures 
of effectiveness, or by both measures of cost and effective- 
ness that are incommensurate Avith one another. In this tech- 
nique, the evaluation criteria underlying the analysis are 
identified at the tops of columns and arranged in decreasing 
importance of criteria from left to right as indicated in 
Figure 6. The alternatives are then listed vertically, with 
the alternative that meets the most significant criterion to 
the greatest extent listed first, and so on. This technique 
is useful when numerous alternatives are being evaluated be- 
cause the technique can be used to eliminrite the less likely 
competitive alternatives and allow attention to be focused on 
the tv/o or three major competitors. Whether the fixed-cost or 
fixed-effectiveness approach is used, the tabular array created 
will be very similar. In the fixed-cost approach, major em- 
phasis is placed on the effectiveness attainable; however, 
cost data indicating ]\o\^ the total cost is divided and phased 
is significant and should be shown. On the other hand, in the 
fixed-effectiveness approach, major emphasis is placed on the 
total cost. The advantage of the tabular display technique 
is that the orderly presentation of alternatives cost and cap- 
ability data is permitted so that their impact on the evalua- 
tion can be readily discerned and discussed along with the sig- 
nificant interrelationships . Conclusions, therefore, can be 
reached by visible traceable means. 



The Model Technique 

The model technique, in which either cost models or 
effectiveness models are created, is usually used when the 
basic differences between the alternative systems are rela- 
tively minor, so as- to permit the valid expression of their 
essential differences by a single parameter (Kazanowski , 1968 , 
pp. 129, 134 § 136). These models may be structured along a 
spectrum of abstraction as shown in Figure 7. First, and "the 
least abstract to be considered is the real-world system, with- 
in which the analyst cannot worJc. 'Next the more abstract mo- 
dels of this real-world are examined. This includes the 
technique of operational exercise, gaming, simulation, and an- 
alytical modeling. ' Last, at the opposite end of the spectrum, 
is total abstraction, within which, again, the analyst cannot 
work. 
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Figure 6, Tabular Display Technique Example 
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GAMING TECHNIQUE 




Figure 7. Model Techniques Spectrum 
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Operational Exercise Technique, Tlic closest approxi- 
mation to the real-world would be an operational exercise, us- 
ing the actual situations with the alternatives under study. 
Operational exercise differs from the real -world in that the 
scale is smaller and many of the personnel are simulated. In 
addition, the consequences of failure are not considered. The 
operation exercise, consequently, is a very close approxima- 
tion to the real alternative but contains certain factors that 
would not normally exist in the real world. The running cost 
of such an exercise is great and the number of alternatives 
which can be analyzed is small. 



Gaming Technique. The next region down this spectrum 
of abstraction is gaming. Here are removed from the represen- 
tation of the real world those components that can most easily 
be simulated by a simple analog such as a computer or some 
other device. 



Simulation Technique. The next stage towards model ab- 
straction is to physically remove the human decision-maker 
from tlie representation* At this point the merits and opera- 
tion of the alternatives are simulated on a computer in which 
the human decision rules are explicitly programmed. 



Analytical Technique. In those rr.odels already consi- 
dered, although modified or compressed, there has been rela- 
tionships between time in the real world and time in the model. 
In analytical models, even this relationship is abandoned. 
Here a system of equations is created: (1) that related the 
characteristics of tlie alternative; and , (2) that are re- 
lated to the measures of effectiveness. Examples of this are 
linear programming models, graphic models, and the wliole class 
of equations referred to as math models. 



Model Considerati ons 

The model spectruju C?igure 7) represents, from top to 
bottom, increasing abstraction from the real-world; or alter- 
nately, proceeding from bottom to top in the figure, increas- 
ing realism or closer approximation to the real world. It is 
also generally observed that both the cost of the analysis and 
thf^ rate at which we can examine different situations decreas- 
es as the model used becomes more abstract. For reasons of 
economy and completeness of analysis, we would like to operate 
as close to total abstraction on this spectrum as possible. 
On the other hand, the possibilities of faulty assumptions in- 
creases as we move away from working in the real world. 



In exploring considerations that go into modeling tech- 
nique selection for a particular cost-effectiveness analysis 
it should be remembered that the basic function of the model 
is to determine the value of the measure of effectiveness 
when the controllable variables of the alternatives are used 
(Blumstein, 1969, pp • 34, 35 § 39). Here the term ''model'' is 
used in a broad sense. Depending on the nature of the goals/ 
objectives, the model used in the cost or effectiveness anal- 
ysis may be formal or informal, very mathematical or not so 
mathematical, heavily computerized, moderately computerized, 
or not computerized at all. The main point, however, is that 
the model need not be highly formal and/or mathematical to be 
useful. The following important points should be considered 
in the selection of modeling techniques. 

• Remember that model building is an art, not a 
science . Of ten it is an experimental process . 

• Emphasize those factors relevant to the goals/ 
objectives , and suppress those factors that 
are relatively unimportant. The model is like- 
ly to be unmanageable if this is not done. 

• Develop a meaningful set of relationships among: 
(1) goals/objectives, (2) the alternatives 
available for attaining the goals/objectives, 
(3) the estimated cost of the alternatives; 
and, (4) the estimated effectiveness for each 
of the alternatives . 

• Base the model design upon the "building-block" 
concept, which accepts analytical data pre- 
pared during prior steps in the creation of a 
set of smaller or partial models. Each small 
model will be used in the construction of a 
larger subsequent cost-effectiveness model 
(Packer, 1968, p. 236) . 

• Make provisions for explicit treatment of un- 
certainty (see Step No. 10) (Fisher, 1967, p. 
72) . 

Within the spectrum of modeling technique (from concrete 
to highly abstract models), there are two techniques that are 
most appropriate for the evaluation of measures of costs and 
measures of effectiveness. It would seem that the choice 
should be dictated by the category of the cost-effectiveness 
study as well as the goals/objectives. This choice is between 
simulation and analytical modeling techniques. Although the 
apparent difference between these two techniques is sometimes 
small, an analytical model is generally more efficient if the 
goals/objectives can be stated in terms such that available 
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mathematical or graphic techniques can be used to solve for 
an optimum solution (Packer, 1968 , 235) • P)Otli analysis of 
cost and effectiveness may make use of analytical models to 
represent: (1) each alternatives being analyzed, (2) its 
operating characteristics; and, (3) the management concepts 
of its operation and logistical support, Tliesc models are 
normally structured so that any parameter, or combination oi 
parameters, can be varied to determine the relative effect on 
the alternatives' total cost and effectiveness (Heaton, 1969, 
p. 35) . 

There are several considerations which should be con- 
sidered in the utilization of analytical models for both cost 
and effectiveness . 



Cost Analytical Model Considerations, The cost model 
relates the goals/objectives/requirements and activity rates 
of the alternative, and of the personnel that operate and/or 
support it, to measures of cost (Bryk, 1965, p, 4). Mathe- 
matical cost models are the most frequently used to evaluate 
— « alternatives when goals/objectives can be mathematically for- 

mulated, when mathematical conditions can be met, and the solu- 
tion can be computed (Miller ^ Rath, 1969, p. 18). The key 
advantage of the use of mathematical cost models is the speed 
with which u number of alternatives can be costed. While ex- 
hibiting significant advantages for specific applications, 
mathematical cost models also possess substantial limitatio.ns . 
^ Two major disadvantages are: 

\ • In creating the cost model a number of impli- 

cit assumptions are made. After the terms of 
tho initial equation are mathematically mani- 
■ pulated and condensed for efficient use, tlie 

initial elements of the cost are no longer 
visible. Thus, the analyst may forget hie in- 
itial assumption and their 1 imitations . 

• Mathematical cost models are relatively in- 
flexible and good basically for one concept 
and its minor variations. If a basic varia- 
tion other then those accounted for by the 
model is to be costed, the model must be re- 
vised and possibly modified. This modifica- 
tion must be verified by comparison with con- 
ventional long-liand results and as a conse- 
quence, the advantage of mathematical cost 
modeling is lost (Kazanowski, 1968, p. 128). 

To conserve time, cost modeling is often performed concurrently 
with effectiveness modeling (Bryk, 1965, p. 14). 

O 
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Effectiveness Analytical tlodel Considerations. Tlic 
effectiveness model relates: (1) the measures of effective- 
ness in achieving the goals/objectives to, (2) measures of 
the alternatives operational performance (Bryk, 1965, p. 2). 
The use of a mathematical effectiveness model is recommended 
v/hen the alternatives under analysis are basically so similar 
that those evaluation criteria that cannot be readily vali- 
dated, can be considered to cancel each other out. This leaves 
only the quantifiable and conunensurable criteria for evalua- 
tion. It should be noted that this occurrence is relatively 
infrequent (Kazanowski , 1968 , p . 129) . 



Analytical Model Computer Use Cons ide rat ions. Analytic 
cost-effectiveness techniques have attained tlicir present level 
of acceptance largely because of the availability of today's 
high-speed computers and the development of appropriate com- 
puter models. The analytical models permit the study of com- 
plete interactions and many alternatives heretofore too com- 
plex and time-consuming, for manual or desk-calculator analy- 
sis (Bell, 1964, p. 2), Prior to these computer programs, the 
cost-effectiveness analyst had to choose among several hand 
models, which were often unwieldy and unrealistic, and the 
older prototype computer models. These older computer models 
v;ere frequently inflexible and failed to give ins ight into 
how conclusions v/ere related to the assumptions (Quade, 196\ 
p. 243) • The computerization of oi^rrent analytical models 
warranted, hov/ever , only v/hen the aaalys is of alternatives in- 
volves repeated computation of complex functions (or when the 
same sub-routines are performed repeatedly). In addition, 
computerized analytical models of advanced future alternatives 
are oftent-^mes of limited value because the expensive program- 
mir.g is time-consuming and soon obsolete by today's standards. 

Concerning cost analytical modeling, if tlie model is 
simple, it does not need to be computerized: if it is com- 
plex, the maintenance of the computer model and associated 
documentation may become time-consuming and expensive. A pos- 
sible compromise might be to computerize those portions of an 
analytical cost model that are not subject to frequent revi- 
sion. In a study directly related to instructional cost model- 
ing, it was indicated that complex mathematical modeling would 
be incompatible with the concept of instructional cost-effec- 
tiveness development. This concept was to develop a procedure 
that was easy to use and not voluminous (Institute for Educa- 
tional Development, 1970, pp. 3-4). Regarding effectiveness 
analytical models, these are seldom mathematical and as a con- 
sequence, computerization is seldom applicable. 



Analytical Model Handbook Considerations. Rather than 
computerize generalized all-purpose analytical models, it lias 
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been found more functional to compile an analytical costing 
and effectiveness handbook. Graphical displays of cost and 
effectiveness relationships, along with their underlying . 
goals/objectives/requirements and assumptions, can be readily 
compiled and utilized. The handbook both allows and makes 
publicly visible the exercise of judgment to ascribe the im- 
pact of subtle deviations from a basic or common design on 
the cost and effectiveness of various alternatives. The cost- 
ing/effectiveness handbook also has the virtue of simplicity, 
speed, flexibility, and economy (Kazanowski, 1968, pp. 131- 
132) . The Institute for Educational Development has developed 
such a handbook, for alternative instructional design s;elec- 
tion, under contract for the Bureau of Naval Personnel and 
naval training schools (Institute for Educational Development,- 
1970, p. 1) . This two-part document is noteworthy in both 
volume and level of detail* Another approach in the develop- 
ment of an analytical model handbook is a document distribu- 
ted by the Department of Defense (Department of Defense, 1969, 
p. 1) . This both outlines the procedure of analysis and en- 
closes analytical forms to be used. 

It should be emphasized the cost-effectiveness models 
can never be completely realistic for they are dependent on 
too many uncertain parameters. The reliance on expert judg- 
ment is indispensable to all analysis. Moreover, the virtue 
of cost-effectiveness analysis is the provisions of a frame- 
work that allows the judgment and intuition of specialists 
in diverse fields to be combined* This framework is the an- 
alytical model which represents a simplified, stylized repre- 
sentation of those aspects of the real world as appropriate 
to the goals/objectives (Quade, 1968, pp. 243, 246-247). 



STEP 9: ANALYZE MERITS OF ALTERNATIVES 

In this final step, the integration of the previously 
derived alternative cost and effectiveness analytical models 
is performed. The purpose of the integration is to combine 
the expected values of alternative cost and effectiveness into 
a single common framework (Seller, 1969, p. 71). This frame- 
work may be based on either the tabular display technique or 
the model technique discussed in Step No. 8* 

An example of the tabular display technique is found in 
Figure 8. This figure is, in addition, an example of the appli 
cation process of cost-effectiveness analysis (previously dis- 
cussed) in whic?i the analysis is both structured and evalua- 
tion IS completed. This display was developed consistent with 
information provided by a cost and effectiveness input simi- 
lar to that found in Figure 6. The instructional designs un- 
der study represents a variety of alternatives for in-service 
employee training. These alternatives are listed in order of 
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RECOMMENDED INSTRUCTIONAL DESIGN FOR EMPLOYEE 
TRAINING IN ORDER OF PREFERENCE ACCORDING TO 
. . COST-EFFECTIVENESS SELECTION CRITERIA 



lilllllllllllllllllliiJIIIIIII^ 


SELECTION CRITERIA 




ALTERNATIVE 
INSTRUCTIONAL DESIGN 


ECONOMY 
FLEXIBILITY 
ADMISSION STANDARDS 
EMPLOYEE ACCEPTABILITY 
EMPLOYER ACCEPTABILITY 
SPEED 


TOTAL NUMERICAL EVAL. 


DAY-RELEASE COURSES 


1 6 5 5 3 6 


26 


FULL TIME SCHOOL 


5 5 4 4 2 1 


21 


SANDWICH COURSES 


.4 4 3 3 4 3 


21 


VOCATIONAL TRAINING 


6 16 6 12 


21 


BLOCK RELEASE COURSES 


3 3 2 2 6 4 


20 


EVENING CLASSES 


2 2 11 5 5 


16 



Note: After Drovet, 1968, p. 221 



Figure 8. Cost-Effectiveness Analysis 
Tabular Display Example 
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recommended preference according to the cost-effectiveness 
selection criteria (Drouet, 1968, p. 221) on an evaluative 
scale of 1 to 6 (6 equals the highest or most preferred) , 

An example of the model technique is presented in Fi- 
gure 9. This example illustrates both a graphic analytical 
model and the procedural process (also previously discussed) 
of cost effectiveness analysis. In this case, the analytical 
model illustrates the logic and procedure of the analysis of 
each alternative instruction design for service training/ The 
model indicates that each alternative instructional design may 
be characterized by: (1) instructional requirements, (2) 
instruction time, (3) course duration, (4) student achieve- 
ment, (5) facility requirements; (6) number of courses per 
year, (7) equipment, (8) course development, (9) instruc- 
tional materials, (lOj software development, and (11) stu- 
dent and instructor travel time. In addition, each instruc- 
tional design can be distinguished by: (1) student achieve- 
ment, (2) student attrition rate, (3) student proficiency 
on the job after graduation, and (4) student morale. Attri- 
butes concerning student performance and morale, during in- 
struction or later, are identified as measures of effective- 
ness. This terminology is used because the effectiveness of 
the instructional design is measured in terms of the end pro- 
ducts; i.e., student course completion.. Other factors, such 
as instructor time per course and amount of instructional ma- 
terials required are classified as measures of efficiency. 
The analytical model indicates that feasibility of the instruc- 
tional design alternatives are determined initially. Next,- 
the characteristics of the feasible alternatives are listed 
as well as the differences in the alternatives' effectiveness, 
efficiency, and costs. Risk factors are also considered in 
the analytical model. Last, the criterion for selecting the 
instructional design is applied (Institute for Educational 
Development, 1969, pp. 23, 25^ 26). This criterion is as 
follows: 

I£ the sum of incremental benefits in dollars of the new 
program exceeds its incremental benefits in dollars of the new 
program [sic, instructional design] .exceeds its increase in 
incremental cost [with acceptable risk], then the new program is 
randed higher than existing program and the existing program is 
eliminated from further consideration. If the reverse is true, 
■the existing program is rated higher and the new program is elim- 
inated [Institute for Educational Development, 1969, p. '29]. 

Regardless of the technique used, both the cost-effec- 
tiveness tabular display technique and the analytical model 
technique relate costs and effectiveness so that the merits of 
the alternatives may be analyzed for the selection of the pre- 
ferred alternative (Bryk, 1955, p. 4) . 
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1^ 1 2 3 4 5 



INPUTS 




INSTRUCTIONAL OBJECTIVES 

• NUMBER AND TYPES OF GRADUATES 

• TYPES OF COURSES 

• STUDENT PROFICIENCY 
•SCHEDULE FOR PRODUCING GRADUATES 

CHANGES IN INSTRUCTIONAL OBJECTIVES 

• INCREASE IN NUMBER OF STUDENTS 
TO GRADUATE 

RESOURCE AVAILABILITY 

• INSTRUCTORS 
•.STUDENTS 

• FACILITIES 

• INSTRUCTIONAL MATERIALS 

• EQUIPMENT 

• COURSE CONTENT DEVELOPMENT 

• SOFTWARE 

• FUNDS 

• TIME 

CHANGES IN RESOURCES AVAILABILITY 
•REDUCTION IN NUMBER OF INSTRUC- 
TORS 



TEST FOR 
FEASIBILITY 
(BASED ON A 
COMPARISON 
OF OBJEC- , 
TIVES WITH 
RESOURCE 
REQUIRE- 
MENTS) 



To 

heet 
No. 2 



ELIMINATE 
INFEASIBLE 
ALTERNA- 
TIVES FROM 
FURTHER 
CONSIDERA- 
TION 



Note: After Institute for Educational Development, 
1969, p. 42 



Figure 9. Cost-Effectiveness Analysis Analytical 
Model Example (Sheet 1 of 5) 
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From 
Sheet^ 
No. 1^ 



I From 
S li e e t 
No. 5 



FEASIBLE INSTRUCTIONAL 
DESIGN ALTERNATIVES 



TOTAL RESOURCE REQUIREMENTS 

• INSTRUCTIONAL TECHNOLOGY 
» STUDENT ATTRITION RATE 

•t COURSE DURATION 

• NUMBER OF STUDENTS 
PER COURSE 

• NUMBER OF COURSES 
PER YEAR 

• INSTRUCTOR TIME PER COURSE 

• INSTRUCTIONAL DESIGN LIFE 

• SCHEDULE 

ADDITIONAL RESOURCE 
REQUI REMENTS 

• INSTRUCTORS 

• STUDENTS 

• FACILITIES 

• INSTRUCTIONAL MATERIAL 

• EQUIPMENT 

• COURSE CONTENT DEVELOPMENT 

• SOFTWARE 

• FUNDS 
§ TIME 



1 



DETERMINE 
DIFFER- 
ENCES IN 
FEASIBLE 
ALTERNA- 
TIVES X 
AND Y 



From 
Sheet 
No. 5 




To 

Sheet 
No. 4 



J NO. b 



Figure 9. Cost-Effectiveness Analysis Analytical 
Model Example (Sheet 2 of 5) 



EFFECTIVENESS MEASURERS 



From 

Sheet 

No. 



INCREASES IN INSTRUC- 

lONAL EFFECTIVENESS 
OF Y OVER X 

REDUCTION IN STUDENT 
INSTRUCTION TIME 
INCREASED STUDENT 
EFFECTIVENESS ON THE 
JOB 

REDUCTION IN STUDENT 
ATTRITION RATE 
IMPROVED STUDENT 
MORALE 

EARLIER AVAILABILITY 
FOR WORK ASSIGNMENT 

INCREASES IN INSTRUC- 
lONAL EFFICIENCY OF 
OVER X 

REDUCTION IN INSTRUC 
TOR COURSE TIME 
REDUCTION IN IN- 
STRUCTIONAL MATER- 
IALS 

REDUCTION IN ADMIN- 
ISTRATION 

REDUCTION IN STUDENT 
TRAVEL TIME AND 
EXPENSES 

REDUCTION IN IN- 
STRUCTOR TRAVEL TIME 
AND EXPENSES 



TRANSLATE 
TO ECONOMIC 
BENEFITS 
AND SUM 
OVER ALTER- 
NATIVE'S 
LIFE 




Figure 9. Cost-Effectiveness Analysis Analytical 
Model Example (Sheet 3 of S) 
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COST 

MEASURES 



INCREASE ON INSTRUC- 
TIONAL CAPITOL COSTS 
OF Y OVER X 

• EQUIPMENT 

• FACILITIES 

• CONTENT DEVELOPMENT 

• SOFTWARE DEVELOP- 
MENT 

• INITIAL SHAKEDOWN 

INCREASE IN INSTRUC- 
TIOr'AL RECURRING 
COST OF Y OVER X 

• RACILITY RENTAL 

• EQUIPMENT RENTAL 

• EQUIPMENT MAINTE- 
NANCE 

• INSTRUCTIONAL 
MATERIALS 



SUM 

INCREMENTAL 
BENEFITS 



To 

Sheet 
No. 5 



CONSIDER 

RISK 

FACTORS 



I 



SUM 

INCREMENTAL 
COSTS OVER 
ALTERNA- 
TIVE'S 
LI FE 



To 

Sheet 
No. 5 



Figure 9. Cost-Effectiveness Analysis Analytical 
Model Example (Sheet 4 of 5) 
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EXCEED THE IN- 
CREASED COST 
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From 
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AS PER 
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Figure 9. Cost-Effectiveness Analysis Analytical 
Model Example (Sheet 5 of 5) 
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Once the alternatives are arranged In order of their 
acceptability in relationship to the evaluation criteria, it 
is f^enerally possible to eliiuxnate the most obviously poorest 
competitors and focus attention on the top three or four. If 
the effectiveness and cost evaluation outcomes for the top al- 
ternative are consistently superior to the respective values 
of the other alternatives, then that alternative is tlie pre- 
ferred selection. If the performance values for the top al- 
ternatives are virtually identical, and no significant differ- 
ence in cost exists, the appropriate conclusion may be that 
there is no significant difference between the top candidates. 
In this case, the adoption of parallel study or development 
efforts of both top candidates may be indicated in order to 
.identify the preferred alternative. This would mean that one 
•alternative would be selected at a later date. If the costs 
of the competing alternatives differ significantly, and the 
evaluation ratings also vary significantly, the selection may 
need to be raade on the basis of personal value judgments 
(Kazanowski, 1968, p. 135). 



STEP 10: ANALYZE UNCERTAINTY FACTORS 

All cost-effectiveness analysis studies include doubt- 
ful features that may not be satisfactorily specified or quan- 
titatively resolved within the study itself (Brechner, 1967, 
p. 57). Important decision problems involve major elements of 
uncertainty. Consequently, a cost-effectiveness analysis of 
such problems must provide for the explicit treatment of un- 
certainty for consideration by the decision-maker. 

Two main types of uncertainty may be distinguished: 

(1) uncertainty about the state of tlie world An the future; 
and, (2) statistical uncertainty about the data on the pre- 
sent state of the alternative . State -of -the -future -uncertain- 
ty stems from chance elements in the real world and would 
exist even if there were no ties of the state to the world of 
the future (Fisher, 1967, pp. 72-73). Statistical data un- 
certainty is most closely associated with cost-effectiveness 
analysis because, unfortunately, the choices between alterna- 
tive candidates are seldom made on the basis of clear-cut data. 
Factors which add to the uncertainties include: (1) alterna- 
tives are frequently inadequate to fully attain the objectives; 

(2) measures of effectiveness may not actually measure the 
extent to which objectives are attained; (3) predictions from 
the cost-effectiveness model are apt to be full of uncertain- 
ties; and, (4) other criteria wliich look almost as attrac- 
tive as the criteria chosen may lead to a different order of 
preference (Quade, 1969, p. 5). Three types of uncertainty 
analysis are most often us to treat statistical uncertain- 
ties. These are : (1) sensitivity analysis; (2) contin- 

^ gency analysis; and, (3) a Fortiori analysis. 

ERIC 
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Sensitivity Analysis ; 

In many instances the output of a cost-effectiveness 
analysis is very sensitive to the assumptions made. The con-* 
elusions reached may be unknowingly yet significantly biased 
by some apparently innocuous assumptions which are essential 
to the analysis (Kazanowski, 1968, p. 138). For example, 
suppose in a given analysis there are several uncertain key 
parameters. In stead of using '^expected values" for these 
parameters, the analyst could test for the influence of these 
assumptions by substituting several values (i.e., high and 
low) in an attempt to see how sensitive the results (the rank- 
ing of the alternatives being considered) are to variations 
in these uncertain parameters (Fisher, 1967, p. 73). Another 
example v/ould be an instance in which the alternatives are 
nearly equal in merit. Here, it would be desirable to attempt 
to-estimate the cost-effectiveness that would be derived from 
the candidate alternatives if they were cancelled at three or 
four different major life-cycle schedule milestones. It is 
also desirable to perform this analysis even v/hen one candi- 
date alternative appears to be clearly superior with respect 
to both cost and effectiveness criteria. Such an analysis may 
point out tlie need for caution in making an otherwise unquali- 
fied endorsement of the alternative (Kazanowski, 1968, pp. 138 
139) ... 



Contingency Analysis 

This type of analysis investigates how the ranking of 
the candidate alternatives maintains ranking when. a relevant 
change in criteria for evaluating the alternatives is postu- 
lated, or a major change in the general environment is intro- 
duced. These techniques assess the degree to which results 
are contingent upon any one factor,- or each of several factors 



A Forti ori Anal ysi s • 

This type of analysis would be applicable when in a 
planning decision problem the generally accepted intuitive 
judgment strongly favors alternative X. The analyst feels, 
however, that X might be a poor choice and that alternative 
Y might actually be preferable. In performing an analysis of 
X versus Y, the analyst may choose deliberately to resolve the 
major uncertainties in favor of X and see how Y compares under 
adverse conditions. 

Although the three types of uncertainty analysis listed 
above may be useful in a direct sense, they may also contri- 
bute indirectly. For example, through sensitivity and 
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contingency analysis the analyst may improve his understanding 
o£ critical uncertainties in a problem area. On the basis o£ 
this knowledge he might then be able to create a new alterna- 
tive that would assume more protection against a wider range 
o£ uncertainties. This is often difficult to do but when it 
can be accomplished, it offers one of the best ways to com- 
pensate for uncertainty (Fisher, 1967, pp. 73-74) • 



STEP 11: DOCUMENT BASES OF PREVIOUS STEPS AND SUBMIT FINDINGS 

The product of a cost -of f ectiveness analysis study, will 
in all probability be a report to the decision-maker. In sup- 
port of this report a key element of systematic cost-effective- 
ness analysis is sufficient documentation of methods, assump- 
tions, sources, etc., so that another analyst would achieve 
substantially the same results witli the same material. With- 
out such documented results, a cost- effectiveness analysis 
appeal for acceptance rests solely on faith in the authority 
and expertise of the analyst with critical examination of the 
way in which he arrived at the recommendations (Heymont, 196 5 
p. 20) . 

Particular emphasis should be placed on the adequacy 
of documenting the following: 

• Specific goals/objectives to be attained. 

• Essential requirements of those goals/objec- 
tives along with associated assumptions • 

• Alternative capabilities and associated assump- 
tions . 

r Alternative costs and associated assumptions 
(learning curves^ time, quantities, etc.) . 

• Alternative evaluation and associated assump- 
tions (scenarios, criteria, etc.). 

• Conclusions: The recommended alternative, the 
limitations, and the associated uncertainty 
factors . 

The usf^ of highly esoteric mathematics for documenta- 
tion should be discouraged. With effort, imagination, and 
forethought, the analyst can usually suitably portray complex 
mathematical and functional relationships in simplified and 
perhaps graphic form. No judicious decision-maker can be ex- 
pected to endorse a conclusion or recommendation whose ration- 
ale and derivation he cannot fully understand. It is the 



responsibility of the analyst to present tlie dociimentcition in 
an appropriate and understandable manner. To have a high pro- 
bability of acceptance by the decision-maker all elements of 
the cost-effectiveness analysis must be documented in such a 
manner that the entire process can be clearly followed (Kaz- 
anowski, 1968, pp. 139-140). 
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SUMMARY 



Cost-effectiveness analysis can be used as an effective 
decision assisting aid: 

• To demonstrate the interactions among the fac- 
tors that characterize alternative courses of 
actions . 

• To enable the consequences of these alterna- 
tive courses to be systematically and objec- 
tively assessed . 

• To promote objective decision-making (Jakobs- 
berg, 1966, p. 42) . 



OPERATIONAL FACTORS OF COST -EFFECTIVENESS ANALYSIS 

Figure 10 was developed to graphically illustrate the 
methods and operational factors available for cost-effective- 
ness analys is . Cost -effectiveness analysis is a methodology 
whereby the choice of one of three broad categories of stu- 
dies may be performed. These are (1) system configuration/ 
component study; (Z"" system comparison study; and, (3) 
suprasystem comparison study. The next involves choice of the 
analytical process to use. This choice is either: (1) the 
procedural process; or, (2) the application process. The 
third decision involved the choice of elements of the stand- 
ard procedure of the actual cost-effectiveness analysis itself. 
This was a choice of analytical techniques between: (1) the 
tabular display technique and, (2) the model technique. 



FUNCTION DIFFERENCEf. AND CHARACTERISTICS OF 
COST-EFFECTIVENESS ANALYSIS 

From the cost-ef fecti v^eness analyst's point of view, the 
selection of study category and analytical process to be used 
usually is determined by requirements outside of the control of 
the analyst. Only within the standard procedure of the cost- 
effectiveness analysis itself does the analyst have a choice in 
analytical technique. Figure 11 was developed to illustrate 
the function, differences, and characteristics of the various 
cost-effectiveness methods. 
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COST-EFFECT- 
IVENESS 
ANALYSIS 
METHODOLOGY 



DECISION 
POINT NO 
1 



DECISION 
POINT NO, 
2 



SYSTEMS CON- 
FIGURATION/ 
COMPONENT 
STUDY CATE- 
GORY 



o 



SYSTEMS 
COMPARISON 
STUDY 
CATEGORY 



SUPRASYSTEMS 
COMPARISON 
STUDY 
CATEGORY 




PROCEDURAL 
ANALYTICAL 
PROCESS 



1 



APPLICATION 
ANALYTICAL 
PROCESS 




DECISION 
POINT NO, 
3 



TABULAR 
DISPLAY 
TECHNIQUE 




MODEL 
TECHNIQUE 



Note: After Pearson, 
1972, p. 108 



STANDARD PROCEDURE 
FOR ANALYSIS 



Figure 10. Cost-Effectiveness Analysis 
Methodological Elements 
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COST-EFFECTIVENESS ANALYSIS STANDARD PROCEDURE 
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Note: After Pearson, 


1972, pp. 112-114 



Figure 11, Cost-Effectiveness Analysis Function, Differences, 
and Characteristics (Sheet 1 of 3) 
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FUNCTION 


DIFFERENCES 


CHARACTERISTICS 


STEP 6 
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Figure 11. Cost-Effectiveness Analysis Function, Differences, 
and Characteristics (Sheet 2 of 3) 
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FIINCTIDN 


DIFFERENCES 


CHARACTERISTICS ■ 


STEP 9 
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OF ALTERNATIVES 
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Figure 11. Cost-Effectiveness Analysis Function, Differences, 
and Characteristics (Sheet 3 of 3) 
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Function 

The following functional elements (in the form of a 
standard step procedure) are necessary in the performance of 
a cost-effectiveness analysis, 

• Step 1: Review definition of general goals/ 
ob j ectives . 

• Step 2: Review identification of parameters 
or requirements . 

• Step 3: Review development of alternatives. 

• Step 4: Establish cost and effectiveness 
evaluation criteria, 

• Step 5: Determine cost of alternatives. 

• Step 6 : Determine capabilities or effective- 
ness approach , 

• Step 7: Select fixed-cost or fixed-effective- 
ness approach, 

• Step 8: Create alternatives versus criterion 
array, 

• Step 9: Analyze merits ^of alternatives, 

• Step 10: Analyze uncertainty factors, 

• Step 11: Document basis of previous steps and 
submit findings. . 

The grouping of the above functional elements inevitably leads 
to some duplication of material, but without the performance 
of these elements, the cost-effectiveness analysis will be of 
lesser quality and the findings questionable. 

Differences 

Differences occur in the performance of various cost- 
effectiveness analysis in several functional elements of the 
standard procedural steps. These differences are found and 
described in some detail in Step 4, Step 8, Step 9, and Step 
10 of .Figure 11, 
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Characteristics 

Characteristics o£ the differences in Step 4, Step 8, 
Step 9, and Step 10 of the functional elements of the stand- 
ard procedure are also considered in Figure 11. It was noted, 
however, that the principle differences in 'tbst -effectiveness 
analysis methods are in Step 8 (create alternative versus cri- 
terion array) and in Step 9 (analyze merits of alternatives) . 
These two differences are actually the same; i,e., the choice 
of using the tabular display technique or the analytical model 
technique* Thus, this is the only methodological choice avail- 
able to the cost-effectiveness analyst not determined by others 
or influenced by outside conditions. 



CONCLUSION 

Cost-.ef fectiveness analysis, in contrast to other manage- 
ment aids, provides answers by a systematic process which is 
reproducible, assessible to critical examination, and readily 
modified as new data becomes available. At the very least, 
cost- effectiveness analysis can supply a means for selecting 
the numerical quantities related to an instructional design 
in a logical and consistent manner. But before quantitative 
ana-lysis can be of assistance to the decision-maker, it must 
be tempered with and used alongside of experience, experimen- 
tation and judgment (Quade, 1969, p. 15). Indeed, cost-effec- 
tiveness analysis cannot be a substitute for judgment. Judg- 
ment must be involved in determining what elements shall be 
considered in the analysis and how they shall enter- There- 
fore, cost-effectiveness analysis supplements judgment by in- 
ductive reasoning which is fully documented. Human judgment, 
however well informed, can still lead to wrong decisions. It 
is believed, however, that cost -effectiveness analysis can at 
least enhance the possibility that the decision is the best, 
one. After an appropriate analysis, the instructional manage- 
ment decision-maker will be in a much better position to make 
a decision--al though he may not find the decision easier to 
make (Herd, 1965, p. 79). 
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APPENDIX 



DEFINITION OF TERMS USED 

Cost. Unit of resources that is the limiting constraint. 
Dollars are used in most cases but other resources, sucli as 
manpower, materials, and facilities will also be considered 
as measurers of cost (llatry, 1969, p. 44). 

Cost-benefit analysis. A problem solving approach 
which requires the definition of objectives, and identifica- 
tion oC the alternative that: (1) yields tlie greatest bene- 
fit for a given cost; or what amounts to tlie same thing, 
(2) that yields a required amount of benefits for the least 
cost. Tlie term usually applies to situations in which tlie 
alternative outputs can be quantified in dollars. A chief 
characteristic is that the aim is to calculate the present 
value of benefits and costs, subject to specified constraints. 

Cost-effectiveness analysis. A problem solving approach 
which requires tne definition of objectives, identification of 
alternative ways of achieving the objective, and identifica- 
tion of the alternative that: (1) yields the greatest effec- 
tiveness for any given cost; or what amounts to the same 
thing, (2) that yields a required degree of effectiveness 
for the least cost. The term is usually used in applications 
in whicli the alternative outputs cannot be easily quantified 
in dollars (McGivney and Nelson, 1969, p. 105). 

Cost-utility analysis. Same as cost-effectiveness an- 
alysis (Anshen, 1967, p. 3) . Numerous terms currently convey 
the same general meaning (i.e., systems analysis, operations 
analysis, cost-effectiveness analysis, and operations research); 
however, they have varying connotations to different people 
(Fisher, 1967, p. 66). Because of such confusion in terminolo- 
gy ai.d meaning, cost -eff ec tivenes^ will be the term used to re- 
flect the level of intent of the study. 

Di scounted cost . An analytical technique that accounts 
for the fact that money to be paid in the future yields in- 
vestment return until the point in time when it is actually 
spent; consequently, present money spent is worth less than 
future money spent (Seller, 1969, p. 17). 

Effective. The accomplishment of the recognized objec- 
tives (Beynon, 1968, p. 84). 



Initial cost. Investment in goods and services re- 
quired to establish and operate a system. Tliese goods liave 
a useful value of longer than a year; thus the costs are 
not repeated every year (Esseff , 1970) • 

Life cycle costs. Combined initial costs and opera- 
tional costs for the estimated useful life of the system 
(Esseff, 1970) . 

Model • Relations used to portray real or expected 
conditions,, actions, or effects in order to predict the out- 
come of actions (Heymont, 1965, p. 56). 

Objective. A statement that describes in observable 
and measurable terms the expected output performance of the 
product of the system (Banathy, 1968, p. 89). 

Planning, programming, and budgeting system (PPBS) . 
Cost-benefit analysis on a large scale (Commission on Instruc- 
tional Technology, 1070, p. 90). A conceptual approach to 
decision-making wliich emphasizes outputs, program activities, 
and accomplishments. Long-range planning, analytic evalua- 
tive tools, and economic rationality are basic ingredients 
(Hartley, 1968 , p. 2 58) . 

System. An integrated relationship of people, equip- 
ment, and methods appropriately organized to accomplish de- 
fined tasks (Heymont, 1965, p. 59). 

Systems analysi^s (systems approach) . Self -correcting 
and logical methodology of decision -making to be used for the 
design development of manmade entities. Strategies of this 
methodology include: (1) the formulation of performance ob- 
jective, (2) the analysis of functions and components; and, 
(3) implementation (Banathy, 1968, p. 91). 

Time-phased cost. The presentation of costs by the time 
period in which the costs occur rather than a total cost 
figure (Heymont, 196 5, p. 60). 

Trade-off. The weighing of alternative means to be em- 
ployed for the accomplishment of required functions (Banathy, 
1968, p. 91). Cost-effectiveness represents one aspect of 
trade-off analysis . 
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